automate building Release Candidate in github action #1391

kevinjqliu · 2024-12-01T19:42:00Z

Related to #1306 and #872
Devlist discussion: https://lists.apache.org/thread/oowhcfwv3fcjzdzm76tbn99k5q84mr75

This PR changes .github/workflows/python-release.yml to build the artifacts for both svn and pypi and update the "how to release doc" to match the new process. Other improvements related to the release process are also included:

Trigger GitHub Action on tag push
Trigger GitHub Action manually with required input fields for both version and rc
Validate version and rc from either tag or manual input
Validate version against the current library version
Changed actions/checkout@v4 fetch-depth from 0 to 1 to fetch only a single commit (fetch-depth 0 fetches all history for all branches and tags)
Generate artifact for SVN named svn-release-candidate-${VERSION}rc${RC}
Generate artifact for PyPi named pypi-release-candidate-${VERSION}rc${RC}, set library version to one with RC (e.g. 0.8.1rc1, same behavior as before)

Final release process should look something like this

prepare local branch for release (main for major/minor, branch for patch release)
Create and push tag with RC (e.g. 0.8.1rc1)
Tag push automatically triggers the GitHub Action
Github action builds artifacts for both svn and pypi
Download svn artifact, create checksum and signature, and upload to SVN dev dist
Download pypi artifact and upload to PyPi
Send vote email
Once passed, download artifact from SVN dev dist, upload to SVN release dist and pypi
If a new RC is needed, create and push another tag with RC (e.g. 0.8.1rc2)

Testing

trigger workflow manually
- trigger workflow manually (version=0.8.0, rc=10, https://github.com/kevinjqliu/iceberg-python/actions/runs/12166623947) ✅
- trigger workflow manually, invalid version (version=0.8.a, rc=10, https://github.com/kevinjqliu/iceberg-python/actions/runs/12167579948) ✅
- trigger workflow manually, invalid rc (version=0.8.0, rc=a, https://github.com/kevinjqliu/iceberg-python/actions/runs/12167639523/job/33936658282) ✅
- trigger workflow manually, input version doesn't match the current library version (version=1.0.0, rc=100, https://github.com/kevinjqliu/iceberg-python/actions/runs/12167654762) ✅
trigger workflow via tag push
- trigger workflow via tag push (tag=pyiceberg-0.8.0rc10, https://github.com/kevinjqliu/iceberg-python/actions/runs/12167691265) ✅
- trigger workflow via tag push, invalid version tag=pyiceberg-0.8.arc10, not triggered ✅ )
- trigger workflow via tag push, invalid rc tag=pyiceberg-0.8.0rca, not triggered ✅ )
- trigger workflow via tag push, input version doesn't match the current library version tag=pyiceberg-1.0.0rc100, https://github.com/kevinjqliu/iceberg-python/actions/runs/12168199006 ✅)

kevinjqliu · 2024-12-05T20:17:21Z

@Fokko could you take a look at this PR when you get a chance?
I tested the new action and instructions via my fork, had to push this branch to main on my fork in order to run the GitHub Action.

Fokko · 2025-02-02T19:28:43Z

.github/workflows/python-release.yml

+  push:
+    tags:
+      # Trigger this workflow when tag follows the versioning format: pyiceberg-<major>.<minor>.<patch>rc<release_candidate>
+      # Example valid tags: pyiceberg-0.8.0rc2, pyiceberg-1.0.0rc1


👍 In line with the current tags: https://github.com/apache/iceberg-python/tags

.github/workflows/python-release.yml

Fokko · 2025-02-02T19:32:27Z

.github/workflows/python-release.yml

+
+      - name: Install Poetry
+        run: |
+          pip install poetry


Should we reuse make install-poetry here? In the Makefile we pin the version of Poetry

good catch! changed

we use make install-poetry in other places for github action
https://grep.app/search?f.repo=apache%2Ficeberg-python&f.repo.pattern=iceberg-python&q=make+install-poetry

Fokko · 2025-02-02T20:27:37Z

.github/workflows/python-release.yml

+          fi
+
+  # SVN
+  svn_build_artifacts:


For consistency, should we use dashes here?

Suggested change

svn_build_artifacts:

svn-build-artifacts:

Same for some other jobs below

Fokko

The flow makes sense to me. Let's give this a shot for the 0.9.0 release 👍

Fokko · 2025-02-02T20:33:03Z

mkdocs/docs/how-to-release.md


-![Github Actions Run Workflow for SVN Upload](assets/images/ghactions-run-workflow-svn-upload.png)
+If `gh` is available, watch the GitHub Action progress using:


kevinjqliu · 2025-02-02T20:50:03Z

had to add an unrelated change in mkdocs/docs/api.md. The url referenced is dead, causing CI to fail

kevinjqliu · 2025-02-02T20:50:42Z

Thanks for the review @Fokko :)

Fokko · 2025-02-03T06:00:28Z

Any time, sorry for taking so long :3

kevinjqliu mentioned this pull request Dec 4, 2024

"how to release", add gh command to trigger workflow #1387

Closed

kevinjqliu force-pushed the kevinjqliu/run-pypi-and-svn branch from b26c2a7 to b22117b Compare December 4, 2024 06:46

This was referenced Dec 4, 2024

[infra] Prevent running python-release action without a specified rc version #1386

Closed

Improve PyIceberg release process #1306

Closed

kevinjqliu requested a review from Fokko December 5, 2024 20:16

kevinjqliu added this to the PyIceberg 0.9.0 release milestone Feb 1, 2025

Fokko reviewed Feb 2, 2025

View reviewed changes

.github/workflows/python-release.yml Show resolved Hide resolved

Fokko reviewed Feb 2, 2025

View reviewed changes

kevinjqliu added 20 commits February 2, 2025 11:50

run two builds together

3ffd8e0

sha and gpg

1866261

import gpg key

1eb1758

add version and rc as inputs

99a18bd

add input validation

ab830e3

add examples

cd7e8e6

fail if not valid

94decda

reorder operations

26462e4

use bash for windows

3b7b097

download/upload

67773fc

name

957548d

only include relevant files

c75a354

rename

b78cf84

use env

f14ac9e

rename

87ef036

fetch only latest

6924864

remove comments

1994c67

remove svn & gpg

a4e0651

add tag trigger

8a37da5

bug

c216260

kevinjqliu added 10 commits February 2, 2025 11:50

pass variables correctly between jobs

e028e44

correctly use rc

fb76e74

env

4af5928

lint

8156488

add rc validation

9ca7550

add docs

e938f90

add gh commands

bf1cb69

use subshell

3e9d9e4

release docs first

695dbd3

make install-poetry

e0e68a3

kevinjqliu force-pushed the kevinjqliu/run-pypi-and-svn branch from e60ab87 to e0e68a3 Compare February 2, 2025 19:52

Fokko reviewed Feb 2, 2025

View reviewed changes

Fokko approved these changes Feb 2, 2025

View reviewed changes

kevinjqliu added 2 commits February 2, 2025 12:40

_ -> -

49cb146

daft dead link

5bd598e

kevinjqliu merged commit fa78256 into apache:main Feb 2, 2025
3 checks passed

kevinjqliu deleted the kevinjqliu/run-pypi-and-svn branch February 2, 2025 20:50

Fokko mentioned this pull request Feb 3, 2025

[infra] clean up cibuildwheel build #1599

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

automate building Release Candidate in github action #1391

automate building Release Candidate in github action #1391

Uh oh!

kevinjqliu commented Dec 1, 2024 •

edited

Loading

Uh oh!

kevinjqliu commented Dec 5, 2024

Uh oh!

Fokko Feb 2, 2025

Uh oh!

Uh oh!

Fokko Feb 2, 2025

Uh oh!

kevinjqliu Feb 2, 2025 •

edited

Loading

Uh oh!

Fokko Feb 2, 2025 •

edited

Loading

Uh oh!

Fokko left a comment

Uh oh!

Fokko Feb 2, 2025

Uh oh!

kevinjqliu commented Feb 2, 2025

Uh oh!

Uh oh!

kevinjqliu commented Feb 2, 2025

Uh oh!

Fokko commented Feb 3, 2025

Uh oh!

Uh oh!


		![Github Actions Run Workflow for SVN Upload](assets/images/ghactions-run-workflow-svn-upload.png)
		If `gh` is available, watch the GitHub Action progress using:

automate building Release Candidate in github action #1391

automate building Release Candidate in github action #1391

Uh oh!

Conversation

kevinjqliu commented Dec 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu commented Dec 5, 2024

Uh oh!

Fokko Feb 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko Feb 2, 2025

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko Feb 2, 2025

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Feb 2, 2025

Uh oh!

Uh oh!

kevinjqliu commented Feb 2, 2025

Uh oh!

Fokko commented Feb 3, 2025

Uh oh!

Uh oh!

kevinjqliu commented Dec 1, 2024 •

edited

Loading

kevinjqliu Feb 2, 2025 •

edited

Loading

Fokko Feb 2, 2025 •

edited

Loading